Market Basket Analysis Algorithm on Map/Reduce in AWS EC2

As the web, social networking, and smartphone application have been popular, the data has grown drastically everyday. Thus, such data is called Big Data. Google met Big Data earlier than others and recognized the importance of the storage and computation of Big Data. Thus, Google implemented its parallel computing platform with Map/Reduce approach on Google Distributed File Systems (GFS) in order to compute Big Data. Map/Reduce motivates to redesign and convert the existing sequential algorithms to Map/Reduce algorithms for Big Data so that the paper presents Market Basket Analysis algorithm with Map/Reduce, one of popular data mining algorithms. The algorithm is to sort data set and to convert it to (key, value) pair to fit with Map/Reduce. Amazon Web Service (AWS) provides Apache Hadoop platform that provide Map/Reduce computing on Hadoop Distributed File Systems (HDFS) as one of many its services. In the paper, the proposed algorithm is executed on Amazon EC2 Map/Reduce platform with Hadoop. The experimental results show that the code with Map/Reduce increases the performance as adding more nodes but at a certain point, Map/Reduce has the limitation of exploring the parallelism with a bottle-neck that does not allow the performance gain. It is believed that the operations of distributing, aggregating, and reducing data in the nodes of Map/Reduce should cause the bottle-neck.